use:
mex -compatibleArrayDims rcmgmex_com_dists.c rcmg_com_dists.c

to main compability with 32 bit, see mex -setup for details.